{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f0d56746-8ec3-4e0c-9b29-523cbcbae2de",
   "metadata": {},
   "source": [
    "### 【A/B测试】用户点击率分析实战\n",
    "\n",
    "https://www.heywhale.com/mw/project/64be8acdb5d3cee03110b076/content"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f82aa948-456f-4246-9c25-302d52be3417",
   "metadata": {},
   "source": [
    "#### 基本概念"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aed638b1-8dd4-4161-8049-3f5d5cae5e16",
   "metadata": {},
   "source": [
    "- AB测试（A/B测试）是一种实验设计方法，用于比较两个或多个版本的产品或策略（比如：两个或多个版本的页面或广告进行测试），并确定哪个版本更有效或更受欢迎。AB测试通常在市场营销、用户体验优化和产品开发等领域广泛应用。\n",
    "\n",
    "- AB测试的基本原理是将用户随机分成两组（A组和B组），每组用户分别接触不同的产品或策略，然后收集数据并进行分析，以确定哪个版本更好。在AB测试中，A组是控制组，用于对照参照，不进行任何变化；B组是实验组，应用新的变化或策略。\n",
    "\n",
    "- AB测试的优势在于能够基于实际数据做出决策，避免凭主观感觉或直觉进行决策，从而更加客观和科学。它可以帮助企业有效地优化产品或策略，提高用户体验和业务效果。然而，AB测试也需要注意一些注意事项，例如样本量的选择、测试时间的长短等，以确保测试结果的可靠性和有效性。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "557b2e73-af83-44f7-9dd9-b714542dc84e",
   "metadata": {},
   "source": [
    "#### AB测试的步骤"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18c00af0-9888-40bb-933e-f221d49901b2",
   "metadata": {},
   "source": [
    "1. 目标设定：明确测试的目标和假设，例如提高转化率、增加点击次数等。本次实验目标就是为了提高点击率\n",
    "\n",
    "2. 分组和随机化：将用户随机分为A组和B组，确保两组之间的差异是随机的，以避免偏见。（本次数据集已完成随机分组）\n",
    "\n",
    "3. 实验设计：设计A组和B组的不同变量，例如网页的不同设计、广告的不同文案等。（本次数据集中test即实验组已设置某一特性）\n",
    "\n",
    "4. 实施实验：将A组和B组的不同版本同时推出，确保测试条件相同。\n",
    "\n",
    "5. 数据收集：收集用户的行为数据，例如点击率、购买量等。 \n",
    "\n",
    "6. 数据分析：对收集到的数据进行统计学分析，比较A组和B组的差异，判断哪个版本更好。在A/B测试中，常用的检验方法有以下几种：z检验、t检验、配对t检验、方差分析、非参检验、比例检验等。\n",
    "\n",
    "7. 结论和优化：根据数据分析的结果得出结论，并根据需要优化产品或策略。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9dd2bd3a-edac-43d9-b528-e1aa847cb394",
   "metadata": {},
   "source": [
    "#### 常见A/B test检验方法"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02103ffe-d835-4bf9-a5fe-596f586c0862",
   "metadata": {},
   "source": [
    "1. z检验（Z-test）：适用于比较两个样本均值的差异，假设数据满足正态分布并已知总体标准差。\n",
    "\n",
    "2. t检验（T-test）：适用于比较两个样本均值的差异，假设数据满足正态分布，但总体标准差未知。\n",
    "\n",
    "3. 配对t检验（Paired T-test）：适用于比较两个相关样本的均值差异，例如同一组用户在不同条件下的测量值。\n",
    "\n",
    "4. 方差分析（ANOVA）：适用于比较多个组别之间的均值差异，例如同时测试多个版本的效果。\n",
    "\n",
    "5. 非参数检验：适用于数据不满足正态分布的情况，例如Wilcoxon秩和检验、Kruskal-Wallis检验等。\n",
    "\n",
    "6. 比例检验：适用于比较两个样本比例的差异，例如点击率、转化率等。（例检验为非参检验，无需要求样本为正态分布，故不做正态检验。）\n",
    "\n",
    "选择适当的检验方法取决于数据类型、样本量、假设条件以及实验设计。在进行A/B测试时，需要根据具体情况选择合适的检验方法，以确保检验结果的可靠性和有效性。同时，还需要注意样本量的大小和实验的时间长度，以避免得出虚假或不准确的结论。在进行统计检验前，最好对数据进行预处理，确保满足检验的前提条件，例如正态分布、独立性等。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "00b26e1e-ce9d-4bcf-9005-dab7b6ea4273",
   "metadata": {},
   "outputs": [],
   "source": [
    "import scipy.stats as stats\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfec3ae1-0023-4a60-97f6-bce580c04d69",
   "metadata": {},
   "source": [
    "本数据集为模拟数据，汇总了用户在A/B测试中网页浏览和按钮点击的数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b22b895c-d987-4c1c-9192-09d283b57c31",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>group</th>\n",
       "      <th>views</th>\n",
       "      <th>clicks</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>control</td>\n",
       "      <td>10.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>control</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>control</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>control</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>control</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id    group  views  clicks\n",
       "0        1  control   10.0     0.0\n",
       "1        2  control    1.0     0.0\n",
       "2        3  control    1.0     0.0\n",
       "3        4  control    2.0     0.0\n",
       "4        5  control    3.0     0.0"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"ab_test_results.csv\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e6ffc7ef-e18b-4861-9994-80470f08e467",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['control', 'test'], dtype=object)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.group.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "908919f1-d006-4fec-943b-70e16c63f75a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 120000 entries, 0 to 119999\n",
      "Data columns (total 4 columns):\n",
      " #   Column   Non-Null Count   Dtype  \n",
      "---  ------   --------------   -----  \n",
      " 0   user_id  120000 non-null  int64  \n",
      " 1   group    120000 non-null  object \n",
      " 2   views    120000 non-null  float64\n",
      " 3   clicks   120000 non-null  float64\n",
      "dtypes: float64(2), int64(1), object(1)\n",
      "memory usage: 3.7+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "4e23345e-1f9e-4e86-ae6a-398cf6f45145",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>group</th>\n",
       "      <th>views</th>\n",
       "      <th>clicks</th>\n",
       "      <th>click_rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>control</td>\n",
       "      <td>297144.0</td>\n",
       "      <td>10303.0</td>\n",
       "      <td>0.034673</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>test</td>\n",
       "      <td>301785.0</td>\n",
       "      <td>11620.0</td>\n",
       "      <td>0.038504</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     group     views   clicks  click_rate\n",
       "0  control  297144.0  10303.0    0.034673\n",
       "1     test  301785.0  11620.0    0.038504"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "click_count = df.groupby(\"group\")[['views','clicks']].sum().reset_index()\n",
    "click_count['click_rate'] = click_count['clicks']/click_count['views']\n",
    "click_count"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fe16385-223a-414d-917d-2b15041181d9",
   "metadata": {},
   "source": [
    "实验组点击率高于控制组，但两组之间是否存在差异，需要通过统计学方法进行检验。本数据集主要对比的差异为两个样本比例的差异，这里选用比例检验最为合适"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30a326d4-c0a7-45e9-9927-946dd9e9d02f",
   "metadata": {},
   "source": [
    "#### 具体检验步骤"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "591e660b-4d27-4878-9a16-226f0737397b",
   "metadata": {},
   "source": [
    "1. 建立假设：首先，我们需要明确研究的问题，并建立原假设（H0）和备择假设（H1）。\n",
    "\n",
    "```\n",
    "H0：两个版本的点击率相等（没有显著差异）。\n",
    "H1：两个版本的点击率不相等（有显著差异）。\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1364fbcf-beb9-440e-9ffa-d5886b8f5685",
   "metadata": {},
   "source": [
    "2. 收集数据：收集两个版本的点击数据，包括广告展示次数和点击次数。通常，我们需要计算每个版本的点击率（点击次数除以展示次数）。\n",
    "```\n",
    "控制组：3.4673%，实验组：3.8504%\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "265ac00f-a80b-4438-bb4b-639d898d1c87",
   "metadata": {},
   "source": [
    "3. 选择合适的比例检验方法：比例检验可以采用卡方检验（Chi-Square Test）或者Z检验（Z-test）来进行。选择方法取决于数据的特点和样本量。如果样本量较大（通常要求每个组别的样本量都大于30），可以使用Z检验；如果样本量较小，可以使用卡方检验。\n",
    "```\n",
    "本数据集样本量较大，可采用z检验\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0f86fa6-b625-46dc-afd7-e6acf3ae33cd",
   "metadata": {},
   "source": [
    "4. 计算检验统计量：根据所选的检验方法，计算比例差异的检验统计量。例如，Z检验可以计算比例之差的标准差，并计算Z值；卡方检验需要构建列联表并计算卡方值。\n",
    "```\n",
    "Z检验计算比例之差的标准差，然后计算Z值。\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65d0c9d6-8b9f-4e3a-b706-6af68c299b24",
   "metadata": {},
   "source": [
    "5. 确定显著性水平：设定显著性水平（通常为0.05），表示在5%的置信水平下，我们能够接受或拒绝原假设。\n",
    "```\n",
    "设定显著性水平为0.05。\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f26cbad-288a-45fe-b773-7193317ec8ca",
   "metadata": {},
   "source": [
    "做出结论：根据计算得到的检验统计量和显著性水平，比较P值（或者查表得到临界值），如果P值小于显著性水平，则拒绝原假设，认为两个版本的点击率有显著差异；如果P值大于显著性水平，则接受原假设，认为两个版本的点击率没有显著差异。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "e09db5b9-af77-4c0b-82e6-742a38c6457c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "比例之差: -0.0038\n",
      "Z值: -7.8935\n",
      "在显著性水平0.05下，拒绝原假设，即两个样本的比例有显著差异。\n"
     ]
    }
   ],
   "source": [
    "p1 = click_count['click_rate'][0]\n",
    "p2 = click_count['click_rate'][1]\n",
    "\n",
    "p_diff = p1 - p2\n",
    "p_pool = click_count['clicks'].sum() /click_count['views'].sum()\n",
    "\n",
    "# 计算标准差 sqrt(p(1-p)*(1/n1+1/n2))\n",
    "SE = np.sqrt(p_pool * (1 - p_pool) * (1/click_count['views'][0] + 1/click_count['views'][1]))\n",
    "\n",
    "# 计算Z值\n",
    "Z = p_diff / SE\n",
    "\n",
    "# 显示结果\n",
    "print(f\"比例之差: {p_diff:.4f}\")\n",
    "print(f\"Z值: {Z:.4f}\")\n",
    "\n",
    "# 设置显著性水平（通常为0.05）\n",
    "alpha = 0.05\n",
    "\n",
    "# 查找临界值（双尾检验）\n",
    "critical_value = stats.norm.ppf(1 - alpha / 2)\n",
    "\n",
    "# 判断是否拒绝原假设\n",
    "if abs(Z) > critical_value:\n",
    "    print(\"在显著性水平0.05下，拒绝原假设，即两个样本的比例有显著差异。\")\n",
    "else:\n",
    "    print(\"在显著性水平0.05下，不能拒绝原假设，即两个样本的比例没有显著差异。\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51f5e52e-8639-4f2c-b1fa-1eba6adc9c2e",
   "metadata": {},
   "source": [
    "本次A/Btest结果，实验组点击率高于对照组，且具有显著差异，验证实验组策略的有效性"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8b4a490-270d-4e23-86f9-bf5509f54b60",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.19"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
